The present invention relates to a method and apparatus for evaluating the decoding of video signals by a xe2x80x9cblack boxxe2x80x9d video decoder. More particularly, this invention relates to an automated method and apparatus that evaluates the functionality of a xe2x80x9cblack boxxe2x80x9d video decoder without the need of a human observer to judge the quality of the decoded video signals.
The increasing development of digital video/audio technology presents an ever increasing problem of resolving the inter-operability of equipment of different manufacturers. Although various organizations, such as the Moving Picture Experts Group (MPEG), created various ISO/IEC international Standards, e.g., 11172 and 13818 (generally referred to as MPEG-1 and MPEG-2), to establish a standard coding/decoding strategy, these MPEG standards only specify a general coding methodology and syntax for generating an MPEG compliant bitstream. Thus, many variations are permitted to accommodate different applications and services. This flexibility creates product differentiation in the marketplace, where it is difficult to ascertain the compliance and quality of an alleged compliant device, e.g., MPEG-like decoders.
Unlike analog television systems, digital decoders are highly non-linear and often contain memory. A digital decoder may operate normally over a certain range of a certain parameter, but may fail dramatically for certain other values of that parameter. Furthermore, a digital decoder may depend on previously decoded data for proper behavior.
Typically, the testing of complex digital systems such as decoders is performed by stimulating the decoder under test (DUT) with a known sequence of data, and then analyzing the output data sequences or the intermediate data sequences using, e.g., a logic analyzer, to determine if the results conform to expectations. Although this is an effective testing technique, it requires extensive knowledge of the circuit implementation or observation of internal nodes of the particular decoder. In fact, a human observer is typically needed to evaluate the output from the decoder under test to determine its compliance.
However, in many instances the decoder is a xe2x80x9cblack-boxxe2x80x9d that accepts a bitstream (encoded signal) as input and provides a digital or analog representation of the decoded signal as an output. Due to product differentiation in the marketplace, it may not be possible to acquire such technical information for all decoders. In fact, even if such technical information is available, it may not be cost effective to construct a different test sequence for every decoder.
Therefore, a need exists in the art for a method and apparatus for testing MPEG-like decoders without prior knowledge of the particular circuit implementation of any particular decoder. More specifically, a need exists for an automated method and apparatus that evaluates the functionality of a xe2x80x9cblack boxxe2x80x9d video decoder without the need of a human observer to judge the quality of the decoded video signals, e.g., whether a DUT is in compliance with a particular standard.
The present invention is a method and apparatus for creating and applying an encoded bitstream that represents one or more test frames, to evaluate the performance of a black box video decoder. More specifically, the encoded bitstreams are designed such that the results from the DUT can be evaluated without the need of a human observer. The present method applies the encoded bitstream to both a DUT and a compliant reference decoder and compares the results from both decoders to determine compliance of the DUT. The present automated method evaluates the differences of the decoded luminance and chrominance signals between the DUT and the reference decoder on a frame by frame basis, which may include one or more levels of analysis, e.g., pixel by pixel, block by block, macroblock by macroblock or etc.
Furthermore, the present encoded bitstream is designed to effect easy frame and/or pixel synchronization of the DUT and the reference decoder. Namely, the synchronization ensures that a comparison is performed on the same frame at the same horizontal and vertical position. This synchronization is achieved through the use of xe2x80x9ckey blocksxe2x80x9d, which are blocks having a specific image pattern.
In one embodiment, the encoded bitstream is designed such that only a subset of the frames within the encoded bitstream is actually compared, thereby increasing efficiency and speed of the test and reducing the storage requirement necessary to store a large quantity of video frames.